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(57) A lane inoage processing system for a vehicle to 
recognize the lane on which the vehicle travels. The sys- 
tem has a CCD camera mounted on the vehicle for view- 
ing a roadway scene ahead of the vehicle to output im- 
age data of the roadway scene including the lane on 
which the vehicle travels. Obstacles such as preceding 
vehicle are detected and a region for image data 
processing is determined (SI 6) so as to avoid the posi- 
tion of the obstacle and the lane on which the vehicle 
travels is recognized based on the processed result. 
More specifically, lane boundary positions estimated up 
to a preceding cycle are stored as historical information 
(S14) and in the region Hough transformation is limited 
in areas determined based on the historical information 
and the horizon in the roadway scene. 
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Description 

This Invention relates to a system for processing 
lane images for a vehicle, more particularly to a system 
for processing lane images for a vehicle enabling a ve- 
hicle to accurately identify vehicle lanes while traveling 
on an expressway or in some other such environment 
including multiple vehicle lanes. 

One existing technology relating to the processing 
of lane images for a vehicle is the method described in 
U.S. Patent No. 4,970. 653, for determining which re- 
gions of images received via a TV camera or the like re- 
quire processing. As shown in Figure 55, in this prior art 
the lane boundaries including lane boundary segments 
that have been lost sight of are actively determined 
based on lane boundary positions determined from an 
earlier image and the region of Hough space to be inves- 
tigated is determined based on the lane boundaries de- 
termined in the preceding frame. 

Japanese Laid-open Patent Application No. Sho 
62- 155 140 also teaches a method in which lane bound- 
ary positions are estimated from the processing results 
of the preceding cycle, while Japanese Laid-open Patent 
Application No. Hei 1-1 61 403 proposes a method for us- 
ing vehicle movement information to estimate lane 
boundary positions In the currently input image from the 
positions according to the processing results in the pre- 
ceding cycle. 

The above and various other technologies so far 
proposed in connection with lane recognition and image 
processing are likely to err in lane recognition in actual 
traffic conditions when, for example, an obstacle such as 
a preceding vehicle comes into the camera field at a po- 
sition overlapping a region to be processed, as shown in 
Figure 56. 

A first object of this invention is therefore to over- 
come this problem. 

Further, Japanese Laid-open Patent Application No. 
Hei 4-36B7B teaches a technique for detecting a vehicle 
lane (road). This prior art enables detection of the pres- 
ence of a lane to the outside of a broken line insofar as 
the broken line can be recognized. Since perfect recog- 
nitbn of a broken line as such is not possible under all 
driving conditions, however, uncertainty remains at the 
time of the lane pattem changes (see Figure 57). 

A second object of the invention is therefore to pro- 
vide a system for processing lane images for a vehicle 
capable of responding flexibly to changes in the number 
of lanes even when the number thereof is unknown. 

According to a first aspect thereof, the present in- 
vention provides a system for processing lane images 
for a vehicle, including an image sensor mounted on the 
vehicle for viewing a roadway scene ahead of the vehicle 
to output image data of the roadway scene including a 
lane on which the vehicle travels, image data processing 
means for processing the image data to output lane in- 
fornnation. and lane recognition means for recognizing 
the lane on which the vehicle travels based on the output 



lane information. The characteristic feature of the system 
is that obstacle position det rmining means is provided 
for determining the position of an obstacle present in the 
roadway scene, and that processing region detenmining 

s means is provided for determining a region of the data 
image to be processed at least based on the detected 
obstacle position such that the regbn excludes the de- 
tected obstacle position. 

The present invention thus provides a system for 

10 processing lane images for a vehicle which enables sta- 
ble and reliable vehicle self-navigation by setting the re- 
gions of the image to be processed such that the lane 
recognition is unaffected by the presence in the image 
of a preceding vehrcle or other such obstacle and by se- 

is lecting the optimum lane in a road environment in which 
obstacles such as other vehicles are present. 

The foregoing and other objects and advantages of 
the inventbn will be more apparent from the following 
description of a preferred embodiment of the invention. 

20 which is given by way of example only and with reference 
to the accompanying drawings, in which: 

Figure 1 is an overall perspective view for explaining 
a system for processing lane images for a vehicle 
2S according to the invention; 

Figure 2 is a block diagram showing particulars 
regarding the sensors of Figure 1 and the process- 
ing of the sensor outputs; 

30 

Figure 3 is a block diagram similar to that of Figure 
2 for explaining the configuratbn of Figure 2 more 
in temns of function; 

55 Figure 4 is a block diagram showing the image 
processing hardware of the block diagram of Figure 
2 (particulars of the system for processing lane 
images for a vehicle); 

40 Figure 5 Is a main flowchart showing the operations 
of the system for processing lane images for a vehi- 
cle according to the Invention; 

Figure 6 is an diagram for explaining the step in the 
45 nnain routine of Figure 5 lor reading estlnnated vehi- 
cle nnovement data; 

Figure 7 is a flowchart of a subroutine of the main 
routine of Figure 5 for reading lane boundary posi- 
5D Xions and storing the lane boundaries on the vehicle 
side as historical Information; 

Figure 6 is a diagram for explaining the estimated 
Ian boundaries in the "actual plane" referr d to In 
55 the subroutine of Figure 7; 

Figure 9 is a diagram for explaining the rul for deter- 
mining the vehicle position in the subroutine of Flg- 
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ure 7; 

Figure 10 is a diagram for explaining the determina- 
tion of start points and end points in the step in the 
subroutine of Figure 7 for determining the start s 
points and end points of approximative straight lines 
from estimated lane boundary data in the form of 
multipoint lines; 

Figure 11 is a diagram for explaining the estimated "io 
lane boundaries on the vehicle side and the selected 
start and end points; 

Figure 1 2 is a diagram for explaining the step in the 
subroutine of Figure 7 for using estimated locus data 75 
at the time of most recent image Input and the esti- 
mated locus data at the time of linage input In the 
current cycle to correct the start points and end 
points to the positions at the time of image input in 
the current cycle; 20 

Figure 13 is diagram for explaining the step in the 
subroutine of Figure 7 for calculating the slopes and 
intercepts of the straight lines from the corrected 
start point and end point positions; 2S 

Figure 14 is a flowchart of a subroutine of the sub- 
routine of Figure 7 for calculating each (p, 6) in the 
Image from the slope of each line in the actual plane 
and storing the result as historical information; so 

Figure 15 is a diagram for explaining the subroutine 
according to the flowchart of Figure 1 4; 

Figure 1 6 is a diagram for explaining camera param- 3S 
eters used in the subroutine according to the ftow- 
chart of Figure 14; 

Figure 1 7 is a flowchart of a subroutine of the main 
routine of Figure 5 for reading distance to an obsta- ^0 
cle; 

Figure 18 is a diagram for explaining the step in the 
subroutine of Figure 17 for using estimated locus 
data at the time of obstacle detection and estimated 
locus data at the time of Image input in the current 
cycle to correct obstacle positions to positions at the 
time of image input in the current cycle; 

Figure 19 is a diagram for explaining an image ^0 
inspection zone used in the step of the subroutine 
of Figure 17 for determining the presence of obsta- 
cles In the camera field; 

Figure 20 is a diagram for xplaining similar det r- 55 
mination in the camera field in th actual plane; 

Figure 21 Is a diagram for explaining the step In the 



subroutine of Figure 17 for converting th positions 
of obstacles in the camera field from coordinates in 
the actual plane to coordinates In the Image; 

Figure 22 is a flowchart of a subroutine of the main 
routine of Figure 5 for setting the processing regions 
In the current cycle based on historical information 
and obstacle positions; 

Figure 23 is a flowchart of a subroutine of the sub- 
routine of Figure 22 for setting the processing 
regions when no historical Information Is available; 

Figure 24 is a diagram for explaining why the middle 
portions of the processing regions set by the sub- 
routine of Figure 23 overlap; 

Figure 25 Is a diagram for explaining the positions 
of the processing regions set by the subroutine of 
Figure 23; 

Figure 26 is a diagram for explaining processing 
regions set by the subroutine of Figure 23; 

Figure 27 Is a flowchart of a subroutine of the sub- 
routine of Figure 22 for setting processing forbidden 
regions when historical information is not available 
but obstacle data are available; 

Figure 26 is a diagram for explaining the subroutine 
according to the flowchart of Figure 27; 

Figure 29 is flowchart of a subroutine of the flowchart 
of Figure 22 for setting the processing regions when 
historical information is available but obstacle data 
are not available; 

Figure 30 is a diagram for explaining the subroutine 
according to Fjgure 29; 

Figure 31 is a flowchart of a subroutine of the flow- 
chart of Figure 22 for finding the nearest obstacle in 
the vehicle's lane; 

Figure 32 is a flowchart of a subroutine of the flow- 
chart of Figure 22 for setting the processing regions 
when historical Information and data on obstacles in 
the vehicle's lane are available; 

Figure 33 Is a diagram for explaining the subroutine 
according to Figure 32; 

Figure 34 is a flowchart of a subroutine of the flow- 
chart of Figure 22 for setting processing forbidden 
regions when historical information and data on 
obstacles outside the vehicle's lane ar available; 

Figure 35 Is a diagram for explaining the subroutine 
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according to Figure 34; 

Figure 36 is a diagram for explaining the l^asis of the 
step in the main routine of Figure 5 for specifying 
Hough transformation constraints using historical s 
information; 

Figure 37 is a flowchart of a subroutine of the main 
routine of Figure 5 for specifying Hough transforma- 
tion constraints using historical infomnation; . io 

Figure 38 is a diagram for explaining the subroutine 
according Figure 37; 

Figure 39 is a diagram outlining the subroutine 
according Figure 37; 

Figure 40 is a graph showing the Hough transforma- 
tion results when the method according to the sub- 
routine of Figure 37 was applied to the road image 20 
of Figure 36; 

Figure 41 is a diagram for explaining the step in the 
main routine of Figure 5 for selecting representative 
lines from the Hough transf ornnation results by clus- 2S 
tering; 

Figure 42 is a flowchart of a subroutine of the main 
routine of Figure 5 for selecting representative lines 
from the Hough transformation result by clustering; 30 

Figure 43 is a diagram outlining the subroutine 
according Figure 42; 

Figure 44 is a diagram for explaining the basis of the 3S 
step in the main routine of Figure 5 for checking for 
misdetection of the boundaries of the vehicle's lane 
using road width; 

Figure 45 is a diagram outlining the step in the main 40 
routine of Figure 5 for using road width to check for 
misdetection of the boundaries of the vehicle's lane; 

Figure 46 is a diagram for explaining the basis of the 
step in the main routine of Figure 5 for checking the 
matching betvyeen representative segments and an 
edge image; 

Figure 47 is a diagram for explaining ideal and actual 
curve recognition, assuming the basis of the step In so 
the main routine of Figure 5 for checking the match- 
ing between representative segments and an edge 
image; 

Figure 48 is a diagram outlining the step in the main ss 
routine of Figure 5 for checking the matching 
betw en representative segments and an edge 
image; 



Figure 49 is the first half of a flowchart of a subrou- 
tine explaining the step in the main routine of Figure 
5 for checking the matching between representative 
segments and an edge image, the explanation 
centering on comparison between candidate seg- 
ments and the edge image; 

Figure 50 is the second half of the f towchart of Figure 
49; 

Figure 51 is a diagram for explaining the subroutine 
according to Figure 49; 

Figure 52 is also a diagram for explaining the sub- 
routine according to Figure 49; 

Figure 53 is a diagram outlining the step in the main 
routine of Figure 5 for checking the matching 
between representative segments and an edge 
image; 

Figure 54 is a diagram for explaining the conversion 
of lane boundary candidate point positions from 
coordinates in the image to coordinates In the actual 
plane conducted as part of the step In the main rou- 
tine of Figure 5 for checking the matching between 
representative segments and an edge image; 

Figure 55 is a diagram for explaining image process- 
ing according to the prior art; 

Figure 56 Is a diagram for illustrating a defect of the 
prior art of Figure 55 and explaining a first object of 
the invention; and 

Figure 57 is a diagram for illustrating a defect of the 
prior art and explaining a second object of the inven- 
tion. 

Figure 1 shows an overall perspective view of the 
system for processing lane images for a vehicle accord- 
ing to the inventbn and loaded on a vehicle. As shown 
in this figure, the system Is equipped with an image sen- 
sor such as a CCD (charge-coupled devbe) mono- 
chrome TV camera 10. The CCD camera (image sensor) 
10 is mounted above the driver's seat In the vicinity of . 
the rearvlew mirror to capture a monocular view of the 
road ahead. The system is further equipped with a radar 
system 12 comprising of 2 millimeter-wave radars 
mounted at the front of the vehicle to detects the pres- 
ence of obstacles such as vehicles and other solid ob- 
jects from the waves they reflect. A yaw rate sensor 14 
Is mounted near the center of the passenger connpart- 
ment for detecting the angular acceleration of the vehicle 
around the gravitational axis (z-axis). In addition, a vehi- 
cle speed sensor 16 constituted as a r ed switch is 
mounted in th vicinity of the vehicle driv shaft (not 
shown) for detecting the vehicle travel speed, and a 
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steering angle sensor 1 8 is provided in the vicinity of the 
steering shaft 20 for detecting the steering angle. 

The steering shaft 20 is equipped with a steering an- 
gle control motor 22, the throttle valve (not shown) is 
equipped with a throttle actuator 24 constituted as a step- 
ping motor, and the brake system (not shown) is 
equipped with a brake pressure actuator 26 (not shown 
in Figure 1). With this configuration, the vehicle navi- 
gates automatically by controlling the steering angle In 
accordance with a calculated steering angle control 
amount, adjusting the vehicle speed by controlling the 
throttle valve opening, and, when necessary, applying 
the brakes. 

The aforesaid configuration is shown In detail in the 
block diagram of Figure 2. The output of the CCD camera 
10 is sent to image processing circuitry or hardware 30 
where straight line segments are extracted by edge de- 
tection and Hough transformation, and the result Is for- 
warded through a bus 32 for storage in a common mem- 
ory 34. An image processing GPU 36 extracts lane 
boundary candidates from among the straight line seg- 
ments and stores them in the common memory 34. An 
image evaluatbn CPU 38 reads the stored values at pre- 
scribed times and uses them to determine the lane 
boundaries. The output of the radar system 12 Is sent 
through a radar processing circuit 40 and the bus 32 to 
be stored In the common memory 34. A radar evaluation 
CPU 42 reads the stored values at prescribed times for 
detecting the positions of obstacles within a coordinate 
system. 

The outputs from the vehicle speed sensor 16 etc. 
are sent to a locus estimating CPU 44 for estimating the 
vehicle locus. An action plan decision making CPU 50 
produces a desired course based on the stored values. 
The desired course and the estimated vehicle locus are 
sent to a locus tracking control CPU 46 which decides a 
locus (desired course) tracking control amount. In addi- 
tion, the locus tracking control CPU 46 calculates the 
steering angle control amount and forwards It to a steer- 
ing angle control CPU 52 which drives the steering angle 
control motor 22 through a PWM (pulse width modula- 
tion) controller 54 and a driver 56. The amount by which 
the motor is driven Is detected by an encoder 58 and 
used for feedback control. 

A speedAracking control section of the action plan 
decision making CPU 50 calculates a desired vehicle 
body acceleration and forwards it to a vehicle speed con- 
trol CPU 60. The vehicle speed control CPU 60 drives 
the throttle actuator 24 through an accelerator stepping 
motor controller 62 and a driver 64 and drives the brake 
pressure actuator 26 through a brake solenoid controller 
66 and a driver 68. The amount by which the brake pres- 
sure actuator 26 Is driven is detected by a pressure sen- 
sor 70 and used for secondary feedback control. 

Although omitted from the drawings in th inter st . 
of simplicity, the configuration also includ s wavefonm 
shaping circuits and other sensor output processing cir- 
cuits. Figure 3 shows th functions of the block diagram 



8 

of Figure 2. Figure 4 is ia block diagrarri showing the im- 
age processing circuitry or hardware of the block dia- 
gram of Figure 2 (the particulars of the system for 
processing vehicle lane images according to the Inven- 
s Won). The Image processing circuitry or hardware 30 of 
Figure 4 specifically includes image input digitizing hard- 
ware 30a, real-time edge detectfon hardware 30b. 
Hough transformation hardware 30c and an image data 
bus controller 30d. 

The operation of the system for processing lane Im- 
ages for a vehicle will now be explained with reference 
to the flowchart of Figure 5. 

First, estimated vehk:le movement data are read in 
S10. 

As shown in Figure 6. this is done by, for example, 
defining the system startup time as time 0 and estimating 
the movement of the vehble In an absolute coordinate 
system whose origin is the position of the vehicle at this 
time. The vehicle's position (x, y) and its yaw angle 6 rel- 
ative to the axis of the coordinate system at time T are 
calculated by a locus estimating section (the locus esti- 
mating CPU 44 in Figure 2) using the speed and yaw 
rate detected by the vehicle sensors. Prior to image In- 
put, the image processing CPU 36 receives these data 
and converts them to Information regarding the vehicle 
position and yaw angle at the time of image input. 

Next, in SI 2, a camera pitch angle correction value 
is calculated from the amount of vehicle pitch angle var- 
iation. 

This is for use In correcting the position and eleva- 
tion of the CCD camera 10, which change owing to 
change in the attitude (pitch angle) of the vehicle during 
acceleration or deceleration. Specifically, this Is 
achieved by mounting height sensors at the fourcomers 
of the vehicle (these sensors are omitted from Figures 1 
and 2), determining the positional relationship between 
the vehicle and the road surface, and using the informa- 
tion obtained to correct the camera elevation, thereby 
preventing errors In mapping calculations between coor- 
dinates in the image and coordinates in the actual plane 
coordinates. 

In the following SI 4, the lane boundary positions es- 
timated by the image evaluation CPU 38 up to the pre- 
ceding processing cycle are read and the lane bounda- 
ries on the vehicle sides are stored as historical riforma- 
tion. This information is an important factor in the setting 
of the processing region of the image input in the current 
cycle and the constraints on the straight tines detected 
by Hough transformation. 

This will be explained with reference to the flowchart 
of Figure 7. First, in SI 00 of this subroutine, the estimat- 
ed lane boundary data up to the preceding cycle are read 
from the image data estimation. In other words, the es- 
timated Ian boundaries in the actual plan of an Image 
data evaluation such as shown In Figure 8 are read. 

Next, in S102, a ch ck is made as to whether lane 
boundary data are available. Since the result of this dis- 
crlminatkDn is naturally NO at system startup, the pro- 
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gram advances to S104 in which a flag is reset to 0 to 
indicate "No historical information*. On the other hand, 
if the result of the discrimination in the SI 02 is YES, the 
program advances to SI 06 in which lane boundary data 
for both sides of the vehicle are derived from all evalu- 5 
ated estimated lane boundary point sequence data. The 
estimated lane boundary data evaluated by image data 
evaluation and the vehicle position are assigned IDs 
from the left side In accordance with the rule set out in 
Figure 9. 

Next, in 8108, the start points and end points of ap- 
proximative straight lines are determined from the lane 
boundary point sequence data. As shown in Figure 10, 
this is conducted by selecting the second point of each 
set of estimated lane boundary point sequence data as 
the start point and the third point thereof as the end point. 
Figure 11 shows the estimated lane boundary on each 
side of the vehicle and the selected start and end points. 

In the following S1 10, the estimated locus data at 
the time of most recent image input and the estimated 
locus data at the time of innage input in the current cycle 
are used to correct the start points and end points to the 
positions at the time of image input in the current cycle. 
As shown in Figure 12, the image data evaluation is con- 
ducted by converting the four points Left-Start, Left-End, 
Right-Start and Right-End in the relative coordinate sys- 
t m (Xrl , Yr1 ) whose origin is the positbn of the vehicle 
at the input time T1 of the image which is the most recent 
image processing result used in the evaluation to points 
in the relative coordinate system (Xr2. Yr2) at the input 
time T2 of the image to be processed in the cun-ent cycle. 

In the next S112, the slopes and Intercepts of the 
straight lines are calculated from the corrected start point 
and end point positions. This operation is explained in 
Figure 13. In the following S114, each (rho, 6) In the im- 
age is calculated from the slope and intercept of each 
line in the actual plane and the result is stored as histor- 
ical information. This calculation is conducted by steps 
S200 to S204 of Figure 14. 

To be specific, the left and right straight lines 
Left (slope A_L, intercept B_L) 
Right (slope A_R, Intercept B_R) 
in the actual plane coordinate system having the vehicle 
position at the tirtie of current cycle image input as its 
origin are converted to the lines in the image, the (rho, 
6) values in the Hough transfomiation coordinate sys- 
tem, namely 

Leff (rho_L. e„L) 
Right' (rho_R, e_R) 
ar calculated, and the results are stored as "historical 
information." (As shown in Figure 15, in this embodiment 
the origin of the Hough transformation coordinate system 
in the Image Is defined as the center of the bwer edge 
of the image, the length of a line drawn perpendicular to 
line L from the origin is defin d as rho, and the angle 
thereof measured counterclockwise from the lower edge 
of the Image is defined as 9.) 

Camera parameters such as shown In Figure 1 6 are 



used in converting positions in the actual plan to posi- 
tions in the innage. In this embodiment, camera height H 
= 1150 (mm), camera elevation Th = 0.043 (rad) and lens 
focal distance F = 12 (mm). In addition, the imaging el- 
ement is a 1/2-Inch CCD (512 x 512 dot), and Xa is set 
to 0.0115 (mnn/dot) and Ya to 0.0095 (mm/dot), where 
Xa and Ya are parameters indicating the length which a 
single dot in memory assumes in the image plane of the 
CCD camera 1 0. The program then advances to S1 1 6 In 
the subroutine of Figure 7, wherein a flag Is set to 1 to 
Indicate "Historical information available*. 

Returning to main routine of Figure 5, the distances 
to obstacles are read in S16. The term "obstacle" here 
refers mainly to the preceding vehicle but also includes 
fixed objects. 

The subroutine for this is shown by the flowchart of 
Figure 17. First, in S300, the most recent forward obsta- 
cle data are read from the radar data evaluation (radar 
evaluation CPU 42 in Figure 2). Next, in 8302, a flag Is 
initially reset to 0 to indicate "No obstacle data", where- 
after a check is made In 3304 as to whether or not ob- 
stacle data are available and the number of obstacles is 
greater than zero. 

If the result in S304 is YES, the program moves to 
S306, in which estimated locus data at the time of obsta- 
cle detection and estimated locus data at the time of Im- 
age input in the current cycle are used to correct obstacle 
positions to positions at the time of image input in the 
current cycle. This operation is explained in Figure 18. 
As shown in this figure, this is conducted by converting 
the position in the relative coordinate system (Xrl, Yrl) 
whose origin is the position of the vehicle at the input 
time T1 of the image which is the most recent image 
processing result used in the evaluation to points in the 
relative coordinate system (Xr2. Yr2) at the Input time T2 
of the image to be processed in the current cycle. 

Next, in S30B, obstacles in the camera field are 
found and if any obstacle is present in the camera field, 
a flag Is set to 1 to indicate "Obstacle data available". As 
shown In Figures 19 and 20, the procedure for finding 
obstacles in the camera field involves using an equation 
for converting the four points Pi, P2, P3 and P4 at the 
left and right extremities of the near and far edges of an 
inspection zone of the Image shown In Figure 19 from 
the Image plane to the actual plane and calculating the 
equations y = AL • x + BL and y = AR • x + BR of the 
straight field boundary lines L and R passing through the 
pairs of points on the left and right, as shown in Figure 20. 

In this embodiment, the inspection zone is set to 
have a length extending from 5 m to 80 m forward of the 
vehicle (xN = 5 (m), xF = 80 (m)), The obstacle positions 
(x, y) are substituted into the L and R equations and the 
obstacle Is judged to be within the camera, field if y is 
between the so-obtain d yL and yR and also within th 
length of the inspection zone. 

Next, in S310, the corrected positions of obstacles 
in the camera f i Id are converted from coordinates in the 
actual plane to coordinates In the image. This operation 
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is explained in Figure 21 . The camera parameters men- 
tioned earlier are used in this conversion. (If the result in 
S304 is NO, the program immediately retums to the main 
routine of Figure 5.) 

Next, in SI 8 of the main routine of Figure 5, the 
processing regions in the current cycle are set on the 
innage based on the historical information and obstacle 
positions. 

The subroutine for this is shown by the flowchart of 
Figure 22. First, a check is made in S400 as to whether 
or not historical information is available. If the result in 
S400 is NO, predefined left and right processing regions 
are set in S402. This operation is conducted according 
to S500 and S502 of the flowchart of Figure 23. 

In the flowchart, first, as shown in Figure 24, left and 
right processing regions that overlap in the middle are 
set in S500. The overlapping of the regions in the middle 
Is for enabling obstacle detection even when the vehicle 
straddles a tane boundary. Next, in S502, the set values 
are stored as the processing regions (Default-L, R). As 
shown in Figure 25, each of the left and right processing 
regions is represented to have a start point (sx, sy) at the 
upper left and to extend lateral and vertical distances 
(1x, ly) from the start point. 

Figure 26 shows the processing regions set in the 
foregoing manner. In this embodiment, the nearest side 
of the recognition distance is set at 5 m and the farthest 
Sid at BO m. The positions of the nearest and farthest 
sides in the image is calculated using an equation for 
converting from actual plane positions to image posi- 
tions. In this embodiment, 16-dot overlaps are provided 
to the left and right of the image center line. This is to 
accommodate vehicle tilt relative to the lane boundary 
during ordinary lane change while driving on an express- 
way. Further, for avoiding the effect of edge detection 
processing and other such filtering, 8-dot margins are 
provided at the left and right edges of the image. 

Next, in S404, a check is made as to whether or not 
obstacle data are available. If the result is YES, the pro- 
gram advances to S406 in which processing forbidden 
regions are set with respect to the left and right process- 
ing regions based on the obstacle position. The obstacle 
referred to here is the preceding vehicle or the like read 
In S16 of Figure 5, The subroutine for this processing is 
shown by the flowchart of Figure 27. The operation is 
explained in Figure 28. 

In the subroutine of Figure 27, the obstacle positions 
ar divided into left, right and center zones (S600). the 
nearest obstacle In each zone is selected (S602), and 
processing forbidden regions of prescribed width W are 
set in the respective zones (S604, S606, S608). Since 
no historical information is available, the processing for- 
bidden regions are centered, on the obstacles. 

Again returning to Figure 22, if S400 finds that his- 
torical information is available, a check is made in S40B 
as to wheth r or not obstacle data are available, and if 
the result is NO, the program advances to 8410 in which 
left and right processing regions are set using a prede- 



fined detection distanc and the historical information. 
The subroutine for this processing Is shown by the flow- 
chart of Figure 29 and the operation is explained in Fig- 
ure 30. 

5 In Figure 29, the predefined detection distance and 
the slopes and intercepts represented by lieft and right 
straight lines are used, first to set a left side processing 
region according to the direction of the slope by calcu- 
lating the intersection with the detection distance (S700, 
10 S702. S704 and S706) and then to set a right side 
processing region by similar processing (S708, S710, 
S712 and S714). Since no obstacle is present in this 
case, the set regions are substantially the same as those 
that would be set in S402. 

In Figure 22, if S408 finds that obstacle data are 
available, S412 makes a search for the closest obstacle 
in the vehicle's lane. The subroutine for this operatbn is 
shown by the flowchart of Figure 31 . 

In this subroutine, a counter value is initialized 
(5800), the value is incremented by 1 (3802), the y-axis 
value yn of an obstacle in the camera field is confirmed 
to be less than Ymax (value on the y-axis of the nearest 
obstacle in the vehicle's lane; explained later) (SB04), yn 
is substituted into the historical information linear equa- 
tions to obtain x-axts values Lx, Rx (S806), a check is 
made as to whether or not the x-axis value xn of the ob- 
stacle falls between Lx and Rx (8808) and, if it does, a 
flag indicating that an obstacle is present in the vehtele's 
lane is set (S810). If it does not, values xn and Lx are 
compared (S81 2). whether the obstacle is to the left or 
right of the vehicle's lane is detected from the result and 
a flag is set indicating "Obstacle present outside vehi- 
cle's lane" (8814, SB16, 88 18). 

It is then confirmed that ad obstacles in the camera 
field have been found and, if any was, information on the 
X- and y-axes positions of the nearest object in the vehi- 
cle's lane is stored, whereafter, the points of intersection 
Lxm, and Rxm with the historical straight lines are stored 
(S820, 8822 and 8824). The nearest obstacle is found 
because distant ones do not hinder the detection. 

The program then advances to 8414 in the subrou- 
tine of Figure 22, where it is determined from the afore- 
said flag whether or not an obstacle is present in the ve- 
hicle's lane. If the result is YES, left and right processing 
regions are set in 841 6 using the distance to the nearest 
obstacle in the vehicle's lane and the historical informa- 
tion. The subroutine for this processing is shown by the 
flowchart of Figure 32 and the operatbn is explained in 
Figure 33. 

According to the procedures mentioned from S900 
to S904 of Figure 32, the left and right processing regions 
are set as shown in Figure 33 so as to avoid a region in 
the vicinity of the obstacle, nam ly, a r gion in which th 
obstacle interferes with det ction of the lane boundaries. 
Thus in this way a processing forbidden or prohibited 
i'ang is set betw en the left and right regions with the 
obstacle as a r ference. 

In Figure 22, if S41 4 finds that no obstacle is present 
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in the vehicle's lane, left and right processing regions are 
set in S41 8 using the predefined detection distance and 
historical information, in the same manner as in S410. 

Next, in S420, a check Is made as to whether or not 
an obstacle is present outside the vehicle's lane and, if 5 
the result is YES, the program advances to S422 in which 
processing forbidden regions are set with respect to the 
established right and left processing regions based on 
the positions of obstacles present outside the vehicle's 
lane. The details of this operation are shown in the sub- 
routine flowchart of Figure 34 and explained in Figure 
35. Specifically, a discrimination is made as to whether 
the obstacle outside the vehicle's lane is to the left or 
right (81000), the nearest obstacle is selected (S1002), 
and processing forbidden regions of a prescribed width 
W are set (SI 004, S1 006). 

The program then moves to 820 In the main routine 
of Figure 5, where Hough transformation constraints are 
specified using historical information. This is aimed at 
eliminating the effect of irregularly appearing shadows, 
road markers and the like by using the historical infor- 
mation in image data evaluation to conduct preferential 
segment detection of at (east the estimated lane bound- 
ari s of the vehicle's lane. It is also for detecting new lane 
boundaries as they become visible at the time of lane 
patt rn changes or the like. 

The purpose of this processing will first be explained 
with reference to Figure 36. As illustrated, when an im- 
age of a road including a shadow or mark is processed 
by Hough transformation, the shadow or mark makes it 
difficult to extract the lane boundaries. At the bottom of 
Figure 36 is shown the Hough transfonmation (rho, 6) 
mapping plane obtained when the image is processed. 
It will be noted that the peak corresponding to the bottom 
line on the right side of the vehicle's lane is hard to dis- 
cem. This is because a large number of peaks of about 
the same magnitude are caused by the road markers. 

For overcoming this problem, it suffices to estimate 
the position of the lane boundaries in the current cycle 
from the results in the preceding cycle and obtain Hough 
transformation results only in the vicinity of the corre- 
sponding peaks. However, since tracking only the same 
lane boundaries as in the preceding cycle makes it Im- 
possible to discover new lane boundaries, it becomes 
difficult to cope with lane pattern changes and lane 
branching. The current step Is for overcoming this prob- 
lem. 

The subroutine for this operation Is shown by the 
flowchart of Figure 37. First, in 811 00, a check is made 
as to whether or not historical information is available 
and, If the result is YES, the program advances to 81102 
and 81 104 in vyhich sorting areas for the left and right 
processing regions are specified. These comprise of pre- 
defined fix d areas and variable areas of prescrib d size 
center d on the straight line data of the left and right side 
historical information. This is illustrated in Figure 38. 

In this embodiment, the variable ar as are set to a 
width of rho : 32 [dot], 8:10 [deg] centered on (rho. 8) 



of the historical information. Th fixed areas are set in 
the range of (rho : 100 - 480 [dot], 9 : 98 - 116 [deg]) and 
(rho: 100 - 480 [dot], B : 65 - 82 [deg]). If 81100 finds 
that no historical information is available, two areas of 
predefined range are set in the left and right processing 
regions In 81 1 06 and 81 1 08. Specifically, fixed areas are 
set In the same manner as above, while (rho : 10 - 380 
[dot], e : 110 - 179 [deg]) and (rho : 10 - 380 [dot], 6:1- 
70 [deg]) In place of the variable areas. 

The foregoing will be amplified with reference to Fig- 
ure 39. Sorting areas are set in the Hough transformation 
(rho, 0) plane of the current cycle to a prescribed width 
centered on the historical Information (rho, 9). As illus- 
trated, since new lane boundaries coming Into view can 
be expected to appear at the horizon, i.e. since it is pos- 
sible to discover new road edges at values of 9 In the 
vicinity of 90 [deg], the sorting areas are pemnanently set 
at fixed widths. (As will be explained later, straight lines 
are extracted from the largest to fiftieth largest peaks In 
the Hough transformation-processed variable and fixed 
sorting areas.) 

Figure 40 shows the results of Hough transformation 
when the present method is applied to the road image of 
Figure 36. It will be noted that the peak corresponding to 
the broken line can be easily distinguished. Thus the es- 
timated lane boundaries are preferentially extracted from 
the Hough transformation results using the historical in- 
formation from the image data evaluation and, similarly, 
preferential extraction is also conducted in the vicinity of 
the horizon where the probability of detecting new lane 
boundaries is high. As a result, it Is possible to detect the 
lane in which the vehicle is traveling with high stability 
while also maintaining flexibility in response to changes 
in the lane pattern or In the number of lanes. 

The program proceeds to 822 in the main routine of 
Figure 5 in which the image is input and its edges are 
detected. Specifically, the CCD camera 10 (image sen- 
sor) views a roadway scene in front of the vehicle and 
the analog NTSC video signal output by the CCD camera 
(monochrome TV camera) 1 0 is converted to a 5 1 2 x 51 2 
pixels, 8-blt quantized digital image. The digital Image is 
imparted with a time delay of the required number of lines 
(three lines in this embodiment) and input for edge de- 
tection processing. In this embodiment, a sobel operator 
was used for the edge detection procession. The edge 
detection processing results were output as a 512 x 512 
edge Image represented at an edge intensity of 8 bits 
(256 magnitude levels). No particular preprocessing for 
correction of pixel intensity or noise reduction was con- 
ducted. This is because the straight line extraction capa- 
bility of Hough transformation is very robust to noise and, 
further, because influence from shadows and road mark- 
ers other than lane boundaries is adequately prevented 
by th provision of the constraints at the time the results 
are selected. 

N xt, in S24, th edge points in the processing re- 
gion are Hough transformed. Specifically, in each of the 
left and right processing regions the edge points of the 
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dg detection processed digital image data having 
magnitude exceeding the threshold value SLD of the 
processing region concerned are Hough transformed 
(according to the "Ohtsu Threshold Value Determination 
Method," described, for example, in {mage Analysis 
Handbook . P503 (University of Tokyo Publishing Socie- 
ty). The edge points In the processing forbidden regions 
are not processed. 

Next, in S26, representative lines are selected from 
the Hough transformation results of the left and right re- 
gions by clustering. This Is for picking out the lane bound- 
aries, i.e. the main line segments, by statistically sorting 
the group of detected straight lines obtained by Hough 
transforming a road image such as shown in Figure 41 . 

Specifically, when Hough transformation, which 
amounts to macroscopic straight line extraction process- 
ing, is applied to an actual road image, the sharply de- 
fined, long lane boundaries appear as large peaks, as 
shown at the lower part of Figure 41 , and a large number 
of straight lines are derived and detected. Because of 
this, it is not possible to obtain the desired road lane 
boundaries merely by selecting the peaks in order start- 
ing with the largest. It is therefore necessary to sort the 
group of nnany detected straight lines based on some 
characteristic so as to narrow them down to only the rep- 
resentative straight lines. However, no completely effec- 
tiv method of sorting (clustering) the detected straight 
lin s obtained by Hough transformation can be found 
among the prior art furthermost neighbor method, cluster 
radius deslgnatbn or K-mean methods. 

When these three methods were tested and com- 
pared, the K-mean method was found to produce the 
most natural cluster division results. In addition, with the 
K-mean method the number of clusters can be increased 
or decreased merely by changing the value of K, while 
with the other methods it is not clear how many clusters 
will result until the processing is completed. In view of 
the foregoing, the clustering method for extracting road 
lane boundary values was based on the K-mean method, 
but the number of clusters K thereof was controlled 
based on the standard deviation within the clusters of the 
K number of clusters and the capability to unite proxi- 
mate clusters was added. 

SI 200 to SI 226 of the flowchart of Figure 42 indi- 
cate the operation for clustering the straight line group 
detected by Hough transformation. Figure 43 is a dia- 
gram outlining the method used. The procedure, which 
will not be explained in detail, comprises of evaluating 
the clustering results by the K-mean method based on 
the standard deviations within the individual clusters and 
repeating the processing while increasing/decreasing K 
(number of clusters). 

It is therefore not necessary to know the number of 
Ian s in advance and, moreov r, respons can be made 
flexibly to changes in the lane pattern. As the standard 
deviation used as the yardstick for dust r division and 
the distance threshold valu for uniting proximate clus- 
t rs, there can be values determined by experiment for 



an actual lane boundary. Because of this, it is possible 
to achieve sorting well matched to the purpose. 

The program next advances to S28 in the main rou- 
tine of Figure 5, In which a check is made for misdetec- 
5 tion of the boundaries of the vehicle's lane using road 
(lane) width. 

This is for checking based on the road width whether 
or not any lines difficult to detect, such as broken lines 
and worn solid lineS: were misdetected. Since, as shown 

10 in Figure 44, the peaks of broken lines resulting from 
Hough transfomnation are smaller than those of solid 
lines (less than 50% of those of solid lines), they are dif- 
ficult to distinguish from noise, making them susceptible 
to misdetection. Therefore, as shown in Figure 45, the 

IS straight line nearest the vehtele is found from among 
those selected by clustering, an opposing estimated 
straight line at a distance equal to the prescribed road 
(lane) width is calculated, and a straight line near thereto 
is found by searching the Hough transformation results 

so down to the smaller peaks. 

In this method, if the straight line found has already 
been detected, this means that this lane boundary had 
a fair degree of magnitude in the Hough transformation 
mapping plane. In such case, since the detected straight 

2S line can be considered reliable, the road width in the ac- 
tual plane is calculated from the two straight lines LI and 
L3 and this new road width is used thereafter. The meth- 
od can therefore deal with changes in road width or some 
degree of difference between the actual road width and 

30 the initially set value. 

The program proceeds to S30 of the main routine of 
Figure 5 in which the matching between the representa- 
tive segments and the edge image is checked. 

If the straight lines detected by Hough transforma- 

35 tion are used as they are as the lane boundaries, line 
segments which do not match with the actual lane 
boundaries will be included at branch lanes leading to 
exit ramps and the like. For avoiding this, earlier technol- 
ogies such as the University of Maryland's DARPA road 

40 edge detection image processing method for ALV (Opti- 
cal Engineering, March 1 986, Vol. 25, No. 3, pp. 409 - 
414) have required sequential processing of small 
processing regkxis. 

This has been an obstacle to shortening processing 

4S time because it requires Hough transformation, which 
accounts for a large proportion of the processing time 
even when high speed hardware is employed, to be con- 
ducted sequentially a large number of times. As shown 
in Figure 46, moreover, when small regions are 

so searched, a method capable of coping with broken lines 
has difficulty coping with curves, and vice versa. In ad- 
dition, susceptibility to influence from shadows and road 
markers is high, making matters difficult without macro- 
scopic information. 

55 Whil road edges can b recognized by straight line 
segment approximation in the case of an ideally shaped 
curve as shown at the top in Figure 47. most curves of 
an actual expr ssway are gentle (radius of curvature of 
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400 m or more) so that the image becomes the same as 
one for a straight road, as shown at the bottom in Figure 
47. Since the prior art therefore recognizes cun/ed road 
portions as straight ones, sudden changes occur at the 
time of entering and exiting curves and these become a 5 
cause for steering control delay and instability. For over- 
coming this problem, it is necessary to sequentially in- 
spect the road edges and conduct processing as if trac- 
ing the curves. As was explained earlier, however, when 
sequential inspection type processing using small re- 
gions is employed, it is difficult to cope with broken lines, 
shadows and the like. 

In light of this, Japanese Laid-open Patent Applica- 
tion Nos. Hel 3-1 39706 and Hei 4-36878 teach systems 
in which the effect of shadows and other such noise is 
eliminated by setting small processing regions at posi- 
tions estimated from the processing results in the pre- 
ceding cycle and curves are coped with by edge inspec- 
tion and the use of processing regions divided into to far 
and near sections. However, this method can handle 
only a prescribed number of lanes and is unable to detect 
new lane boundaries and lane branches that come into 
view when changes occur in the lane pattern. 

This embodiment therefore uses the method shown 
in Figure 48. S1 300 to S1 334 of the flowchart of Figures 
49 and 50 indicate the operation for checking candidate 
segments against the edge image based on this method. 
Figures 51 and 52 are diagrams for explaining the sub- 
routine according to Figure 49. Thus, as shown in Figure 
53, the road edges are detected by a heuristic method 
using the processing results of the preceding cycle. As 
shown in Figure 54, the candidate lane boundary point 
sequences are converted from coordinates in the image 
to coordinates in the actual plane. 

As a result of the foregoing, the adverse effect of 
shadows and road markers is reduced, thereby enabling 
highly precise lane boundary edge detection processing. 
Specifically, since the Hough transformation makes it 
possible to obtain the macroscopb straight lines by 
processing the whole of a large region at one time, there 
is no need to limit the number of lanes in advance, so 
that the aforesaid method is not affected by broken lines 
or by breaks at curves and is moreover able to faithfully 
reproduce curves. 

Since the aforesaid method requires only detection 
of the center of edge point distribution on the scanned 
lines and simple rules for shifting the scanning center, it 
does not involve any complex conditional branching. 
Since line segments are obtained by cutting the repre- 
sentative lines selected by clustering after Hough trans- 
formation at their intersections, the ensuing processing 
can be carried out independently for each divided line 
segment, which makes It possible to achieve high 
processing spe d by proc ssing the segm nts In paral- 
lel. The method is less susceptible to horizontal lines re- 
sulting from shadows and th like than is th method of 
investigating and tracking edge points sequentially and 
is able to cope with broken lines and breaks without need 



for complex inspection rules. 

The program proceeds to 832 in the main routine of 
Figure 5 In which the geometric characteristics of the 
candidate line segments are determined. This involves 
determining the geometric characteristics of each seg- 
ment of the lane boundary candidates detected in S30 
with respect to the other such segments and attaching 
the determined characteristics to the output data. These 
characteristics include, for example, the angles at which 
each segment connects with other segments. 

Next, broken line discrimination is conducted in S34. 
This is for increasing road configuration recognition per- 
formance by discriminating between broken and contin- 
uous lines. The method used is described in detail in the 
Applicant's Japanese l^id-open Patent Application No. 
Hei 3-158976 and will not be explained further here. 

Next, in S36, the detected lane boundary data is out- 
put to the image data evaluation section (the image eval- 
uation CPU 38 in Figure 2) and the program returns to 
the beginning of the main routine (to S10). 

As explained in the foregoing, In this embodiment 
image processing regions are set based on historical in- 
formation and obstacle position so that lane recognition 
is not affected by the preceding vehicle and other obsta- 
cles. As a result, lane recognition errors do not occur 
when driving behind another vehicle, for example. In ad- 
dition, since Hough transformation constraints are spec- 
ified using historical information, changes in the number 
of lanes can be coped with even in a driving environment 
with an unknown number of lanes. 

While the foregoing embodiment uses a monocular 
visual sensor, it is alternatively possible to use a binoc- 
ular visual sensor, which can be used to measure dis- 
tance, making it possible to omit the front radars. 



Claims 

1. A system for processing lane images for a vehicle, 
including: 

an image sensor (10) mounted on the vehicle 
for viewing a roadway scene ahead of the vehicle to 
output image data of the roadway scene including a 
lane on which the vehicle travels; 

image data processing means (30) for 
processing the Image data to output lane infomna- 
tion; and 

lane recognition means (36) for recognizing 
the lane on which the vehicle travels based on the 
output lane Information; 

characterized in that: 

obstacle position determining means 
(12,40,42) is provkJed for determining the position 
of an obstacle pres nt in the roadway sc ne; and 

processing region d termining means (30) is 
provided for d termining a region of the data image 
to be processed at least based on the detected 
obstacle position such that th region excludes the 
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detected obstacle position. 

2. A system according to claim 1. wherein said 
processing region determining means (30) deter- 
mines the region such that the region is made up of 
a left half and a right half that overlap partially with 
each other. 

3. A system according to claim 1 or2, further including: 

lane boundary position storing means (34) for 
storing a lane boundary position estimated up to a 
preceding image data processing as historical infor- 
mation; and 

said processing region determining means 
(30) determines the region based on the detected 
obstacle position and the historical information. 

4. A system according to any of preceding claims 1 to 
3. wherein said image data processing means (30) 
includes: 

area determining means for determining an 
area In the image; 

edge image extracting means (30b) for 
extracting edge images in the image data of the 
area; 

Hough transforming means (30c) for carrying 
out Hough transformation on the edge images; 

straight tine detecting means for detecting 
from one from among 

straight lines obtained from the Hough trans- 
formation; 

and 

said lane recognition means recognizes the 
lane on which the vehicle travels based on the 
detected straight line. 

5. A system according to claim 4, further including: 

lane boundary position storing means for stor- 
ing a lane boundary position estimated up to a pre- 
ceding image data processing as historical informa- 
tbn; and 

said area determining means determines a 
first area based on the historical Infomnation and a 
second area based on a position where a new lane 
boundary can be expected to appear. 

6. A system according to claim 5. wherein said area 
determining means determines the second area 
based on a position corresponding to a horizon in 
the roadway scene. 

7. A system according to any of preceding claims 4 to 
6, wherein said straight line detecting means 
Includes; 

setting m ans initially setting centers of clus- 
ters of the straight lines obtained from the Hough 
transformation; 

clustering means for clustering the straight 



lines obtained from th Hough transformation; 

defining means for defining an average in each 
cluster as new centers; and 

uniting means for uniting proximate clusters to 
s detect said straight line. 

a A system according to any of precednng claims 1 to 
6, wherein said Image sensor is a CCD camera (10). 
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FIG. 2 
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both sides of vehicle from 
all evaluated estimated lane 
boundory point sequence date 



I 



sm 



Determine start points and end 
points of approximative straight 
lines from lane boundary point 
sequence data 



I 



SW8 



Use estimated locus data at time 
of most recent Image input and 
estimated locus data at time of 
image input in current cycle to 
correct stort points and end 
points to positions at time of 
image input in current cycle 



sm 

v 



Colculote slopes and intercepts of 
straight lines from corrected stort 
point and end point positions 



V 



_L 



Calculate each (p, e) in image 
from slope and intercept of each 
line in actual plane and store 
result as historical information 
I 



sm 



Set flag to indicote / 
"Historical Information ovallable" V 



sm 



( RETURN^ 
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FIG. 8 



Road edge 



Attribute indicating whether 
broken or solid line 



Road edg 



1 r/f 



Yr 



I 

i> 
I 

9 
I 

4- 



Road edge 2 

Vehicle position at time of 
preceding image input 



Estimated lane boundaries in actual plane 



FIG. 11 



Xr 



Left.End(x.y) 

Left-Start(x,y) 
Yr- 



\ 



Right.End(x,y) 



Right^tart(x,y) 



Estimated lane boundary on each side of 
vehicle and selected start and end points 
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FIG. 14 



( START ) 



Convert slope A and intercept B in actual 
plane to slope A« and Intercept Bi in image 

Y = A^X + B-»y = AiX-x + Bi 



Convert straight line y = Al x + Bi 
in image to (p, 6) in Hough transformation 
coordinate system 



'S200 



■S202 



Store image straight line data(Ai, BI) and 
(p, e) as historical information 



S204 



( RETURN ) 
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FIG. 15 



Straight 
line Left 



Xr 



Yr 



Straight 
line Right 



Actual plane coordinate system 
having vehicle position at current 
cycle Image input time as its origin 



Left side 
straight line 




Right side 
straight line 



Estimated lane boundaries in image 
(Historical Information) 
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FIG. 16 



Lens focal distance F [mm] 



Camera elevation 
Th [rod] 



1 



image plane (CCD) 
Camera height H [mm] 



Road surface 



Imag 

C( 


e pi 

:cd; 


one 









I I 
Xi[mm] 




XpCdot] 



Xa= Xi/Xp [mm/dot] 



26 



EP 0 697 641 A2 



FIG. 17 



( START ) . 



S300 



Read most recent forward obstacle data 
from radar data evaluation 






^^5302 




Reset flag to indicote 
"No obstacle data" 






5304 



'Obstacle data available- 
jind number of obstacles greater^ 
than zero 1 



yes 



5306 



Use estimated locus data at time 
of obstacle detection and estimated 
locus data at time of image input 
in current cycle to correct obstacle 
positions to positions at time of 
image input in current cycle 



■5308 



Find obstacles in camera field and 
if any obstacle is present in 
camera field, set flag to indicate 
"Obstacle data available " 



S310 



Convert coordinate positions of 
obstacles in camera field from 
those in actual plane to those in 

image 



C RETURN ) 
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FIG, 13 



Use estimated locus data at time of 
obstacle detection and estimated locus 
data at time of image input in current 
cycle to correct obstacle positions to 
positions at time of image input in 
current cycle 



Time T2 

Position and orientotion of vehicle 
in obsolute coordinate system 
determined by locus estimation 
tr(x2. y2, 62) 

Time Tl 

Position and orientation of vehicle 
in absolute coordinate system 
determined by locus estimation 
tr(xl. y1. 61) 




Absolute coordinate 
system in actual plane 



28 



EP 0 697 641 A2 



FIG. 19 



Farthest edge of 
decision zone 



Nearest edge of 
decision zone 




Decision zone in image 



FI G. 20 



Obstacle position 
L \ Xr 




Camera field 

Decision 



- — e 



yL (x.y) 



zone 



xN I q..^-. _ — 



Yr. 



pr ^ 



o ? 



yR 



/O 



P3' 



Decision in camera field in actual plane 

# : Obstacle in camera field 

O s Obstacle outside camera field 
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FI6\21 



Xr 



O 

Obst_l(xr,yr) 



O 



Obst.2 



Yr- 



Positions of obstacles 
in image 



Obstacles in camera f ield 
in actual plane 




Conversion to positions 
in image 



FI6.23 



( START ) 



Set left and right processing regions 
within predefined recognition distance 
range so as to overlap at Image middle 






Store values set for left and right 
processing regions as 
Default.L (sx. sy, Ix, ly) 
DefaullLR (sx, sy, Ix, ly) 







'S500 



S502 



( RETURN ) 
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( START ) 



FIG. 22 



S400 




Find nearest obstacle 
in veliicle's lane 



^ ^ S414 
tjbstacle" 
)resent in vehicle's^ 
lane ? 



S4i6 



yes 



Set left and right proce- 
ssing regions using distance 
to nearest obstocle in . 
vehicle's lane and 
historical information 



S402 



Set predetermined left and 
right processing regions 



S410 



Set left and riglit proce- 
ssing regions using prede- 
fined detection distance 
ond historical information 



Set left and right 
processing regions 
using predefined 
detection distonce 
onA historical 
infor mo tlon 




Set processing forbidden regions with 
respect to established right and left 
processing regions based on positions 

of obstacles present outside 
vehicle's lone 



S404 




Set processing forbidden 
regions with respect to 
earlier set left and right 
processing regions based 
on obstacle position 



X RETURN ^ 
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FIG. 25 



(sx, sy) O 
ly 



IX 



FIG. 26 



Farthest side of 
recognition distance — V 

Nearest side of 



1 











Image center 



Left processing region 



Right processing region 
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FIG. 27 



( START ^ 



Use volue xp of Obst_l(xp 
left, right and center zon 


* yp) to divide obstacles into 
es 




^5602 


Select obstacle in each zone with largest yp value 
(nearest obstacle) 




^5604 



With respect to the nearest obstacle Obst_n{xp, yp), set in 
each of the left and right processing regions a processing 
forbidden region of prescribed width W whose lower edge is 
yp and upper edge is the farthest side of the recognition 
distance (limited to within processing region) 





^ — 'S606 


Left zone L 

W't'^/espect to the nearest obstacle Obst_n(xp. yp). set in 
the left processing region a processing forbidden region of 
prescribed width W whose lower edge is yp and upper edge is 
the farthest side of the recognition distance (limited to 
Within processing region) 


1 — ^S6oa 


Right zone R 

With respect to the neare 
the right processing regloi 
prescribed width W whose 
the farthest side of the r 
within processing region) 


fst obstacle Obst_n(xp, yp), set in 
1 a processing forbidden region of 
lower edge is yp and upper edge is 
ecognition distance (limited to 



C RETURN ) 
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FIG. 28 



Zone 

C (Overlapped center portion) 



._ : lli 












1-4- 





^ ^ ImoQe s 
Left processing center R'Snt processing 
region 



Processing 
forbidden region 



region 
Obstacle (xp, yp) 



Farthest side of 
recognition distance 




Left processing 
region 



In this embodiment, W Is calculated 
(using equation referred to earlier) 
as Width in image corresponding to 2m 
(vehicle width plus allowance) at 
pbstocle distance (position of yp 
in actual plane). 



Farthest side of 
recognition distance — 

Nearest side of 
recognition distance 





• 




t 




< 







Left processing 
region 



Image 
center 



Right processing 
region 
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FIG. 29 



( START ^ 

Set left ■ ^ 

processing region 



S702 




S700 
no 



Find xl of point of 
intersection PL(xU yf) 
xl = Yf/AJ. - B_L/A_L 



S704 



L 



Find xl of point of ' 
intersection PL(xl, yn) 
xl = Yn/A_L - B_L/A_L 



sx.L = Default_L.sx 
sy_L = Def ault_L.sy 
lx_L = xl-DefoultJ_.sx + dx 
ly_L = Def<iultJL.ly 



Set right 
processing region 



5706 



S710 

L 




Find xr of point of 
intersection Pr(xr, Yf) 

xr = Yf/AJ^ - B.R/AJI 



S712 



Find xr of point of 
intersection Pr(xr» Yn) 

xr = Yn/A_R - B_R/AJl 



sm 



sx_R = xr-dx 
sy_R = DefaultR.sy 
Ix-R = Default_R.lx-(xr-DefaultJUx) 
ly-R = Def aultR.iy 



( RETURN ) 
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FIG. 30 



Farthest side of yf 
recognition distance - 

Nearest side of Yn 
recognition distance 

Historical information y 
Left side stroight line 
(A.U B-L) 




xl xr 



dx 



Historical information 
Right side straight line 
(A_R, B.R) 



Farthest side of 
recognition distance 



Nearest side of 
recognition distance 



Pi; 

<D| 



dx 



II 



Left processing region Right processing region 

Processing regions when historical 
information is available but 
obstocle dato are not 



Farthest side of 
recognition distance'^J 



Nearest side of Yn 
recognition distance 

Historical information v 
Left side straight line ^ 
(A_L,B-L) 




Historical information 
Right side straight line 
(A_R. B_R) 



Processing regions 
when A-R<0 
In this embodiment, dx is 4 dots. 

For avoidtfig the effect of edge detection processing and 
Icl^l fi^^ nltenng. 8-dot margins are ogain provided at 
the left and right edges of the Image. 
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C START ^ 



FIG. 31 



n=0, 'im\=6y -^Sd00 



n=n+ll S802 




Calculate Lx, Rx by substituting yn of obstacle . 
Obst_n(xn, yn) In camera field into linear historical 
information equations 
Left side straight line Left : 
y = A_L * X + B_L 
Right side straight line Right : 
y = A Jl ♦ X -i- B-R 




5806 



S808 



S8J2 



Ymax = yn, X = xn 
Lxm = lx, Rxm = Rx 

Set flag to indicate 
'Obstacle present in 
vehicle's lane" 




Obstacle found to be 
outside vehicle's lone 
on left 



S8M- 



I 



5816 



Obstacle found 
to be outside 
vehicle's lane 
on right 



Set f laa to indicate 
"Obstacle present outside 
vehicle's lane " 



'S8J8 



a<total number of obstacles 



5820 




in fieldN.?, 
no 

-Obstacle present In^ 
•vehicle's lane 7 — ■ 

yes 



5822 



Image position (xn* yn) of 
obstacle in vehicle's lone 



no 
5824 



Store Obst_M(X, Ymax) of nearest obstacle 
in vehicle s lane. Also store intersection 
points with historical straight line Lxm, Rxm 



^ — . Historical Informotion 

C RETURtO left side straight line 
^ ■■ ^ (A.L. B_L) 




Historical Information 
right side straight line 
(A_R. BLR) 
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FIG. 3 2 



( START ^ 



Define points of intersection 
between y-direction position 
Yobs of Obst_M(Xobs, Yobs) of 
nearest obstocle in vehicle's lone 
and left and right historical 
straight lines as Lxm, Rxm 



■SQOO 



Set left processing region 

sx_L=DefauIt_L,sx 

sy_L=DefauIt_L.sy 
lx_L=Lxm - Dcfault_L.sx + dx 

ly_L=Default_L.ly 



I 



5<702 



Set right processing region 

sx_R=Rxm-dx 
sy_R=DefauU_R.sy 

lx_R=DefauIt_R.Ix -(Rxm;Default_R.sx) 
Iy_R=Default_R.ly 



I 



( RETURN ^ 
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FIG. 53 



Position in image of object 
in vehicle's lane 



Yobs 



Historical information 
left side straight line 
(A_L. B_L) 




Historical information 
right side straight line 
(A.R, B_R) 



Farthest side of 
recognition distance 



Nearest side of 
recognition distance 




Processing regions when historical information and 
data on obstacle in vehicle's lane, are available 



For avoiding the effect of edge detection processing 
and other such filtering, 8-dot morgins are again 
provided at the left and right edges of the image. 
As earlier, dx is 4 dots. 
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F 1 6.34 



( START ^ 



Use value of xp of ObstJ (xp, yp) to divide 
pbstocles into left and right zones. 
(This was done earlier in procedure for 
finding obstocles in vehicle's own lone.) 



S1000 



Select obstacle in each zone with largest 
yp value (nearest obstacle) 



S1002 



'St004 



Left zone L 

With respect to the nearest obstacle Obst_n(xp, yp), 
set in the left processing region a processing 
forbidden region of prescribed width W whose lower edge 
is yp and upper edge is the farthest side of 
the recognition distance (limited to within 
processing region) 



SI006 



Right zone R 

With respect to the nearest obstacle Obst.n(xp. yp), 
set in the right processing region a processing 
forbidden region of prescribed width W whose lower edge 
•s yp and upper edge is the farthest side of 
the recognition distance (limited to within 
processing region) 



C RETURN ^ 
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FIG. 35 



Historical information dx 

Right side straight line. , 

(A_R. B_R) 1^ 



Obstacle outside 
vehicle's lane 



Historical information 
Left side stroight line 
(AX. B_L) 




Left processing 
region 



Right processing 
region 



Processing 
forbidden region 



Farthest side of 
recognition distance 



Obstacle (xp. yp) 




Left processing 
region 



in this embodiment, W is calculated 
(using equation referred to earlier) 
as. width In image corresponding to 2m 
(vehicle width plus allowance) at 
obstacle distance (position of yp 
in actual plane). 
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[Drawbock of prior art] 



FIG. 56 




original image 



Edge Image - 




Hough transformation mapping plane 
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FIG. 37 




( START ) 



Historical 
information 
available ? 




Sim 



no 





fyes r^^^02 


Specify sorting area (predefined fixed area) 
for left processing region (Specify variable 
areo of prescribed size centered on linear 
data (p_L, e.Dof left side historical ' 
information) 




jrSm4 


Specify sorting area (p 
for right processing re 
area of prescribed size 
data (p_R, eiR) of right 
information) 


>redef ined fixed area) 
gion (Specify variable 

centered on linear 

side historical 



SU06 



Set two areas of predefined 
size and position for left 
processing region 



swa 



Set two areas of predefined 
size and position for right 
processing region 



( RETURN ) 
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to 
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FIG. 40 



1 

t 


lag. - 

\ t r 




r- 






1 




0 


^ 90.. 










0 


[deg] 



Hough transformation mapping plane 
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FI G. 41 




Peaks in Hough transformation mopping plane 
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(start ) 



FIG. 4 2 



Initial number of clusters K=4 



I 



Select 4 peaks starting with largest 
as centers of initial clusters 



S1200 
— SJ202 



pn 



=0 I — ■S1204 



Find center nearest pn-th 
peal^ and define It as ele- 
ment of that cluster 

I 



Si206 



S1208 
S1210 



pn=pn-H I 



yes 



In this embodiment, 
min_P is calculated 
for Hough transform 
Dfl„irr.:-,«\:;s« d result in each pro- 
'^^ M nJ^-in",-^ cessing region as 
[JO pn^^bO,?- mln_P=Max peol(*THR 



SI 




no 



Define averoge of elements 
in each cluster as new 
cluster center 



I 



THR was set at 0.47, 
the value obtained 
empiricolly by comparing 
magnitudes of solid 
line and brolcen line peaks 



Check whether centers of all 
of K number of clusters 
coincide with center in preceding 
cycle 



Define peak nearest center of 
cluster having largest standard 
deviation as center of (K + l)-th 
cluster, ond set K = K + 1 




'S12J4 



St2!6 



Check variance inside cluster for all clusters 
and calculate standard deviation for eoch 



Sf2t8 



-S1220 



-All stondard deviations-^^ 
smaller than Lim_M 7-"^ 

yes 



SJ222 



no 



Llm_M = 0.01 
in this embodiment 



Define largest peak In each cluster 
^ representative peak 

I 



S/22^ljnite proximate clusters 
3 



( RETURN ) 



-51224 



In this embodiment, clusters whose 
representatives are less than 
0.15 apart are united 
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F 16.46 




Broken line 



Search should be able to 
jump over breoks 



Incompatible 



\ Situation is mode 
even more difficult 
by shadow or 
the like 




Lane branching 



At break, search should also 
be able to oscertoin corinection 
with other direction 



FIG. 47 




Ideal curve recognition 




Actual curve recognition 
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[Outline of present method procedures] 
processing region 



Rood edge 
point sequence 



FIG. 48 




Representative line segments obtained 
by using processing region frame to 
cut candidate straight lines detected 
by Hough tronsformotion • 




Find point of intersection between 
candidate stroight lines and divide here 




-or eoch divided line 
segment, measure degree 
of matching with edge 
image 




Method of measuring degree 
of matching with edge image 



Scan edge image over given 
width along candidate One 
segment 




Find distribution center 
of edge points at 
scanned line and define 
the same as detected 
point 



Define line segments with 
greater than prescribed 
degree of matchlnq as 
lone boundory candidates 



For detecting curve into the distance, 
find amount of offset from candidate 
line segment of edge point distribu- 
tion center in scan of preceding cycle 
and shift center of scon in next cycle 
by corresponding amount 





Obtain one-dimensional data 
by converting to length in 
actual plane, calculote rotio 
of length of portion whose 
edge points were detected to 
overall length of candidate 
line segment, and define 
result OS decree of motching 
of extracted point sequence 



Output 

point 

sequence 



Yr. 




Offset amount 



If no edge point Is detected, 
return scon center to condi- 
dote line segment 



Convert detected points to coordinates 
in actual plane ond output as arbitrary 
point sequence data according to edge 
point distribution 
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C START ) 



FIG. 49 



S1300 



--Does 9 of Li satisfy~^_ 
-45 rdegKK135 [deg] 7--^^ 




yes 

i Scon verticollTI ' 



Storting from end point side hoving 
larger coordinate (if equal, from 
end point side having smaller x 
coordinate), sequentially obtain 
scon date over of fsettable width 
2W f 1 centered on line segment Li 



'S1306 



Calculate offset of each set of scon 
data as (Sum of coefficients of 
positions where edge points at or 
above threshold value were detected) 
/(Number of flogs nf indicating 
detection of edge points at or 
obove threshold value) 
(Round down decimol fractions) 



S1308 



Define sum of coordinate at 
center of scan data in current cycle 
and omount of offset calculated 
in current cycle as y' and store 
coordinate (x, y') as center of edge 
distribution 



Defme sum of offset from Une 
segment Li m current cycle and 
offset amount colculated in 
current cycle as offset of scan 
data center from Li in next cyde. 
If number of edge detection 
flogs nf is zero, retirn offset 
from Li to zero 
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Scon horjzontolly I 



i 



Starting from end point side having 
larger i coordinate (if equoU from 
end point side having smaller jl 
coordinate), sequentially obtain 
scon data over of fsettable- width 
2W + 1 centered on line segment Li 
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Calculate offset of each set of scan 
data as (Sum of coefficients of 
positions where edge points at or 
above threshold value were detected) 
/(Number of flags nf indicating 
detection of edge points at or 
above threshold value) 
(Round down decimal fractions) 



Define. sum of jc coordinate at 
center of scan data in current cycle 
and amount of offset calculated 
m current cycle as x' and store 
coordinate (x; y ) as center of edge 
distribution 



I 




Define sum of offset from line 
segment Li in current cycle and 
offset amount calculated in 
current cycle as offset of scan 
data center from Li in next cycle. 
If number of edge detection 
flags nf is zero, return offset 
from Li to zero 




canning to other 
end point completed V 



no 
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FIG. 50 



Number of detected edge points 
in each set of scan data 



(A) 



0|2|3|3|0|0| 1 |0|0|2|.1 I 2| 
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Remove isolated noise by applying 
medion filter to one-dimensional 
data obtained by lining up for each 
set of scon doto the numbers of 
times edge points at or obove 
threshold value SLD were detected 



—J In this embodiment, 
the filter size is 5 

I 

After filtering 



11- l2|2h ro]|o|oh h 
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The number of edge ---£303-^ 

points detected in / | ' 
a each scon is 



Calculate (Number of sets of scan 
data in which edge points at or 
above threshold value SLD were 
detected)/(Total number of sets 
of scan data Ns) ond define result 
as degree of matching Et of line 
segment Li with edge image 



^ 

not on issue here, r /y^ 
The Vfllue of n.is r /> i 
either 1 or 0. i ' ■ ' 



□ 




Ns 
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n^ is the sum of number of detected 
edge points, which do not have to 
be continuous. (Because isolated 
noise is removed by the one- 
dimensional median filter) 



51330 



Specify line segment Li as 
itne segment to be output 



SJ332 



SI334 
L 



Remove line segment 
If from line segments 
to be output 



Considering edge point distribution center 
coordinates stored In each set of scan data 
as lane boundary candidate point positions, 
convert them from coordinates in image to 
coordinates in octual plane. (As output 
points to be converted, select only 
points equal or nearest to predefined 
table of y coordinates in image. 
In this ernbodiment, selection is made 
so as to obtain a sequence of points 
spaced about 5m in actuol plane) 



In this embodiment, 
the empirically determined 
value 0J5 was used as min.E* 
This is a level enabling 
detection of even quite badly 
worn broken lines 



y coordinat^ 
table in image 



( RETURN ^ 



Become equally spaced 
in actual plane 



impg 



t> 
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FIG. 51 



Candidate line segment Li(p, 0) 
Slope A, Intercept B 



(x1,y1) 




y Processing region 

W is 4 in this embodiment 
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Position coefficients of scan data 
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FIG. 52 



Flags indicatinci detection of 

O edge point of intensity ot 

or obove threshold value 



O OOO 



In the diagram above, the offset from center is 

(-2 + 1 + 2 +3) / 4 = 1.0 

so that the next scan is offset 1 to the right. 



Candidate line segment 
y = A^x + A 




Edge imoge 
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FIG, 55 



Detected lane boundaries 




Image Input in preceding prior ^r*- 
cycle P'^'or art 



Processing results in 
preceding cycle 




Input in current cycle 



FIG. 56 



Wrong segment detected owing 
to effect of edge of 
preceding vehicle 




Prior ort 



Processing results in 
preceding cycle 




Distance 
information 



Processing region 
according to invention 



R1 
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FIG. 57 




What was being recognized as 
a two-lane road became 
a three-lane road by addition 
of another lane on the right 
side 



Even though the lane pattern has 
changed, the new lane that has 
come into view cannot be detected, 
so that the vehicle cannot move 
to the lane on the right 
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FIG. 5 



Colcuiote comsro pitci: angle correction VQluel — ,S12 
iTom omount of vehicle pitch ondle vgr'totton t 



Reod estimated vehicle movement dotcl SfO 



Read lane boundory positions estimated by image evoluation 
CPU 38 UP to precedino processing cyde and store lone 
boundories on vehicle sides os historical informotion 



Reod distonces to obstoclesH - — S/5 



Set processino regions in current cycle based on 
histoticol information end obstode positions 



Specify Houoh tronsforniatton limits using 
historlcol informotion 



seo 



input imoge end detect edges [ •^--'Sgg 

Carry opt Hough tfmslormotion on edge points in prcccsring reBtonj 



Select representative tines from Kou9b trans- 
formation results of left and right regions by 
clustering • 
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Check for lone boiuidory misdetection using rood (lone) 
width 



Check motcNng between represo\tottve segments ond edge 
inoge. (Calculate degree of (notching with edge iousge and 
define point sequence on line segonent with greater than 
prescribed degree of matching os (one boundary contfiddte 
point sequence. Lone boundory condUate point 
sequences ore converted to coordhiotes in actual plane) 



Determine geometric choracteristics of candidate k 
line segments (Length, ongle) j V 



T 
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S32 . 



Discrimimrte broken lines J 



~S34 



Output detected lone boundory doto] S36 
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